BetaZero: Belief-State Planning for Long-Horizon POMDPs using Learned Approximations

Real-world planning problems\unicode{x2014}including autonomous driving and sustainable energy applications like carbon storage and resource exploration\unicode{x2014}have recently been modeled as partially observable Markov decision processes (POMDPs) and solved using approximate methods. To solve high-dimensional POMDPs in practice, state-of-the-art methods use online planning with problem-specific heuristics to reduce planning horizons and make the problems tractable. Algorithms that learn approximations to replace heuristics have recently found success in large-scale problems in the fully observable domain. The key insight is the combination of online Monte Carlo tree search with offline neural network approximations of the optimal policy and value function. In this work, we bring this insight to partially observed domains and propose BetaZero, a belief-state planning algorithm for POMDPs. BetaZero learns offline approximations based on accurate belief models to enable online decision making in long-horizon problems. We address several challenges inherent in large-scale partially observable domains; namely challenges of transitioning in stochastic environments, prioritizing action branching with limited search budget, and representing beliefs as input to the network. We apply BetaZero to various well-established benchmark POMDPs found in the literature. As a real-world case study, we test BetaZero on the high-dimensional geological problem of critical mineral exploration. Experiments show that BetaZero outperforms state-of-the-art POMDP solvers on a variety of tasks.Comment: 20 page

Changing the script: a typology of Dutch theatre manuscripts in the Southern Low Countries, and the interaction between manuscript and print (Seventeenth–Eighteenth centuries)

NWO016.Veni.195.371Medieval and Early Modern Studie

Reconstruction of three-dimensional porous media using generative adversarial neural networks

To evaluate the variability of multi-phase flow properties of porous media at the pore scale, it is necessary to acquire a number of representative samples of the void-solid structure. While modern x-ray computer tomography has made it possible to extract three-dimensional images of the pore space, assessment of the variability in the inherent material properties is often experimentally not feasible. We present a novel method to reconstruct the solid-void structure of porous media by applying a generative neural network that allows an implicit description of the probability distribution represented by three-dimensional image datasets. We show, by using an adversarial learning approach for neural networks, that this method of unsupervised learning is able to generate representative samples of porous media that honor their statistics. We successfully compare measures of pore morphology, such as the Euler characteristic, two-point statistics and directional single-phase permeability of synthetic realizations with the calculated properties of a bead pack, Berea sandstone, and Ketton limestone. Results show that GANs can be used to reconstruct high-resolution three-dimensional images of porous media at different scales that are representative of the morphology of the images used to train the neural network. The fully convolutional nature of the trained neural network allows the generation of large samples while maintaining computational efficiency. Compared to classical stochastic methods of image reconstruction, the implicit representation of the learned data distribution can be stored and reused to generate multiple realizations of the pore structure very rapidly.Comment: 21 pages, 20 figure

Optimizing Carbon Storage Operations for Long-Term Safety

To combat global warming and mitigate the risks associated with climate change, carbon capture and storage (CCS) has emerged as a crucial technology. However, safely sequestering CO2 in geological formations for long-term storage presents several challenges. In this study, we address these issues by modeling the decision-making process for carbon storage operations as a partially observable Markov decision process (POMDP). We solve the POMDP using belief state planning to optimize injector and monitoring well locations, with the goal of maximizing stored CO2 while maintaining safety. Empirical results in simulation demonstrate that our approach is effective in ensuring safe long-term carbon storage operations. We showcase the flexibility of our approach by introducing three different monitoring strategies and examining their impact on decision quality. Additionally, we introduce a neural network surrogate model for the POMDP decision-making process to handle the complex dynamics of the multi-phase flow. We also investigate the effects of different fidelity levels of the surrogate model on decision qualities

Antitumour and antiangiogenic effects of Aplidin® in the 5TMM syngeneic models of multiple myeloma

Aplidin® is an antitumour drug, currently undergoing phase II evaluation in different haematological and solid tumours. In this study, we analysed the antimyeloma effects of Aplidin in the syngeneic 5T33MM model, which is representable for the human disease. In vitro, Aplidin inhibited 5T33MMvv DNA synthesis with an IC50 of 3.87 nM. On cell-cycle progression, the drug induced an arrest in transition from G0/G1 to S phase, while Western blot showed a decreased cyclin D1 and CDK4 expression. Furthermore, Aplidin induced apoptosis by lowering the mitochondrial membrane potential, by inducing cytochrome c release and by activating caspase-9 and caspase-3. For the in vivo experiment, 5T33MM-injected C57Bl/KaLwRij mice were intraperitoneally treated with vehicle or Aplidin (90 μg kg−1 daily). Chronic treatment with Aplidin was well tolerated and reduced serum paraprotein concentration by 42% (P<0.001), while BM invasion with myeloma cells was decreased by 35% (P<0.001). Aplidin also reduced the myeloma-associated angiogenesis to basal values. This antiangiogenic effect was confirmed in vitro and explained by inhibition of endothelial cell proliferation and vessel formation. These data indicate that Aplidin is well tolerated in vivo and its antitumour and antiangiogenic effects support the use of the drug in multiple myeloma

Extramedullary disease in multiple myeloma: a systematic literature review

Extramedullary involvement (or extramedullary disease, EMD) represents an aggressive form of multiple myeloma (MM), characterized by the ability of a clone and/or subclone to thrive and grow independent of the bone marrow microenvironment. Several different definitions of EMD have been used in the published literature. We advocate that true EMD is restricted to soft-tissue plasmacytomas that arise due to hematogenous spread and have no contact with bony structures. Typical sites of EMD vary according to the phase of MM. At diagnosis, EMD is typically found in skin and soft tissues; at relapse, typical sites involved include liver, kidneys, lymph nodes, central nervous system (CNS), breast, pleura, and pericardium. The reported incidence of EMD varies considerably, and differences in diagnostic approach between studies are likely to contribute to this variability. In patients with newly diagnosed MM, the reported incidence ranges from 0.5% to 4.8%, while in relapsed/refractory MM the reported incidence is 3.4 to 14%. Available data demonstrate that the prognosis is poor, and considerably worse than for MM without soft-tissue plasmacytomas. Among patients with plasmacytomas, those with EMD have poorer outcomes than those with paraskeletal involvement. CNS involvement is rare, but prognosis is even more dismal than for EMD in other locations, particularly if there is leptomeningeal involvement. Available data on treatment outcomes for EMD are derived almost entirely from retrospective studies. Some agents and combinations have shown a degree of efficacy but, as would be expected, this is less than in MM patients with no extramedullary involvement. The paucity of prospective studies makes it difficult to justify strong recommendations for any treatment approach. Prospective data from patients with clearly defined EMD are important for the optimal evaluation of treatment outcomes

MRI in multiple myeloma : a pictorial review of diagnostic and post-treatment findings

Magnetic resonance imaging (MRI) is increasingly being used in the diagnostic work-up of patients with multiple myeloma. Since 2014, MRI findings are included in the new diagnostic criteria proposed by the International Myeloma Working Group. Patients with smouldering myeloma presenting with more than one unequivocal focal lesion in the bone marrow on MRI are considered having symptomatic myeloma requiring treatment, regardless of the presence of lytic bone lesions. However, bone marrow evaluation with MRI offers more than only morphological information regarding the detection of focal lesions in patients with MM. The overall performance of MRI is enhanced by applying dynamic contrast-enhanced MRI and diffusion weighted imaging sequences, providing additional functional information on bone marrow vascularization and cellularity. This pictorial review provides an overview of the most important imaging findings in patients with monoclonal gammopathy of undetermined significance, smouldering myeloma and multiple myeloma, by performing a 'total' MRI investigation with implications for the diagnosis, staging and response assessment. Main message aEuro cent Conventional MRI diagnoses multiple myeloma by assessing the infiltration pattern. aEuro cent Dynamic contrast-enhanced MRI diagnoses multiple myeloma by assessing vascularization and perfusion. aEuro cent Diffusion weighted imaging evaluates bone marrow composition and cellularity in multiple myeloma. aEuro cent Combined morphological and functional MRI provides optimal bone marrow assessment for staging. aEuro cent Combined morphological and functional MRI is of considerable value in treatment follow-up

A pattern-search-based inverse method

Uncertainty in model predictions is caused to a large extent by the uncertainty in model parameters, while the identification of model parameters is demanding because of the inherent heterogeneity of the aquifer. A variety of inverse methods has been proposed for parameter identification. In this paper we present a novel inverse method to constrain the model parameters (hydraulic conductivities) to the observed state data (hydraulic heads). In the method proposed we build a conditioning pattern consisting of simulated model parameters and observed flow data. The unknown parameter values are simulated by pattern searching through an ensemble of realizations rather than optimizing an objective function. The model parameters do not necessarily follow a multi-Gaussian distribution, and the nonlinear relationship between the parameter and the response is captured by the multipoint pattern matching. The algorithm is evaluated in two synthetic bimodal aquifers. The proposed method is able to reproduce the main structure of the reference fields, and the performance of the updated model in predicting flow and transport is improved compared with that of the prior model.The authors gratefully acknowledge the financial support from the Ministry of Science and Innovation, project CGL2011-23295. The first author also acknowledges the scholarship provided by the China Scholarship Council (CSC [2007] 3020). The authors would like to thank Gregoire Mariethoz (University of New South Wales) and Philippe Renard (University of Neuchatel) for their enthusiastic help in answering questions about the direct sampling algorithm. Gregoire Mariethoz and two anonymous reviewers are also thanked for their comments during the reviewing process, which helped improving the final paper.Zhou ., H.; Gómez-Hernández, JJ.; Li ., L. (2012). A pattern-search-based inverse method. Water Resources Research. 48(3):1-17. https://doi.org/10.1029/2011WR011195S117483Alcolea, A., & Renard, P. (2010). Blocking Moving Window algorithm: Conditioning multiple-point simulations to hydrogeological data. Water Resources Research, 46(8). doi:10.1029/2009wr007943Alcolea, A., Carrera, J., & Medina, A. (2006). Pilot points method incorporating prior information for solving the groundwater flow inverse problem. Advances in Water Resources, 29(11), 1678-1689. doi:10.1016/j.advwatres.2005.12.009Arpat, G. B., & Caers, J. (2007). Conditional Simulation with Patterns. Mathematical Geology, 39(2), 177-203. doi:10.1007/s11004-006-9075-3Caers , J. 2002 Geostatistical history matching under training-image based geological model constraintsCaers, J. (2003). Efficient gradual deformation using a streamline-based proxy method. Journal of Petroleum Science and Engineering, 39(1-2), 57-83. doi:10.1016/s0920-4105(03)00040-8Caers, J., & Hoffman, T. (2006). The Probability Perturbation Method: A New Look at Bayesian Inverse Modeling. Mathematical Geology, 38(1), 81-100. doi:10.1007/s11004-005-9005-9Capilla, J. E., & Llopis-Albert, C. (2009). Gradual conditioning of non-Gaussian transmissivity fields to flow and mass transport data: 1. Theory. Journal of Hydrology, 371(1-4), 66-74. doi:10.1016/j.jhydrol.2009.03.015Carrera, J., & Neuman, S. P. (1986). Estimation of Aquifer Parameters Under Transient and Steady State Conditions: 1. Maximum Likelihood Method Incorporating Prior Information. Water Resources Research, 22(2), 199-210. doi:10.1029/wr022i002p00199Chen, Y., & Zhang, D. (2006). Data assimilation for transient flow in geologic formations via ensemble Kalman filter. Advances in Water Resources, 29(8), 1107-1122. doi:10.1016/j.advwatres.2005.09.007Christiansen, L., Binning, P. J., Rosbjerg, D., Andersen, O. B., & Bauer-Gottwein, P. (2011). Using time-lapse gravity for groundwater model calibration: An application to alluvial aquifer storage. Water Resources Research, 47(6). doi:10.1029/2010wr009859De Marsily, G., Delay, F., Gonçalvès, J., Renard, P., Teles, V., & Violette, S. (2005). Dealing with spatial heterogeneity. Hydrogeology Journal, 13(1), 161-183. doi:10.1007/s10040-004-0432-3Deutsch, C. V., & Tran, T. T. (2002). FLUVSIM: a program for object-based stochastic modeling of fluvial depositional systems. Computers & Geosciences, 28(4), 525-535. doi:10.1016/s0098-3004(01)00075-9Dubuisson, M.-P., & Jain, A. K. (s. f.). A modified Hausdorff distance for object matching. Proceedings of 12th International Conference on Pattern Recognition. doi:10.1109/icpr.1994.576361Emsellem, Y., & De Marsily, G. (1971). An Automatic Solution for the Inverse Problem. Water Resources Research, 7(5), 1264-1283. doi:10.1029/wr007i005p01264Evensen, G. (2003). The Ensemble Kalman Filter: theoretical formulation and practical implementation. Ocean Dynamics, 53(4), 343-367. doi:10.1007/s10236-003-0036-9Falivene, O., Cabello, P., Arbués, P., Muñoz, J. A., & Cabrera, L. (2009). A geostatistical algorithm to reproduce lateral gradual facies transitions: Description and implementation. Computers & Geosciences, 35(8), 1642-1651. doi:10.1016/j.cageo.2008.12.003Fernàndez-Garcia, D., Illangasekare, T. H., & Rajaram, H. (2005). Differences in the scale dependence of dispersivity and retardation factors estimated from forced-gradient and uniform flow tracer tests in three-dimensional physically and chemically heterogeneous porous media. Water Resources Research, 41(3). doi:10.1029/2004wr003125Feyen, L., & Caers, J. (2006). Quantifying geological uncertainty for flow and transport modeling in multi-modal heterogeneous formations. Advances in Water Resources, 29(6), 912-929. doi:10.1016/j.advwatres.2005.08.002Fu, J., & Gómez-Hernández, J. J. (2008). A Blocking Markov Chain Monte Carlo Method for Inverse Stochastic Hydrogeological Modeling. Mathematical Geosciences, 41(2), 105-128. doi:10.1007/s11004-008-9206-0Jaime Gómez-Hernánez, J., Sahuquillo, A., & Capilla, J. (1997). Stochastic simulation of transmissivity fields conditional to both transmissivity and piezometric data—I. Theory. Journal of Hydrology, 203(1-4), 162-174. doi:10.1016/s0022-1694(97)00098-xGuardiano, F. B., & Srivastava, R. M. (1993). Multivariate Geostatistics: Beyond Bivariate Moments. Geostatistics Tróia ’92, 133-144. doi:10.1007/978-94-011-1739-5_12Harbaugh , A. W. E. R. Banta M. C. Hill M. G. McDonald 2000 MODFLOW-2000, the U.S. Geological Survey modular ground-water model-User guide to modularization concepts and the ground-water flow process Reston, Va.Hendricks Franssen, H. J., & Kinzelbach, W. (2008). Real-time groundwater flow modeling with the Ensemble Kalman Filter: Joint estimation of states and parameters and the filter inbreeding problem. Water Resources Research, 44(9). doi:10.1029/2007wr006505Franssen, H.-J. H., Gómez-Hernández, J., & Sahuquillo, A. (2003). Coupled inverse modelling of groundwater flow and mass transport and the worth of concentration data. Journal of Hydrology, 281(4), 281-295. doi:10.1016/s0022-1694(03)00191-4Henrion, V., Caumon, G., & Cherpeau, N. (2010). ODSIM: An Object-Distance Simulation Method for Conditioning Complex Natural Structures. Mathematical Geosciences, 42(8), 911-924. doi:10.1007/s11004-010-9299-0Hoeksema, R. J., & Kitanidis, P. K. (1984). An Application of the Geostatistical Approach to the Inverse Problem in Two-Dimensional Groundwater Modeling. Water Resources Research, 20(7), 1003-1020. doi:10.1029/wr020i007p01003Hoeksema, R. J., & Kitanidis, P. K. (1985). Analysis of the Spatial Structure of Properties of Selected Aquifers. Water Resources Research, 21(4), 563-572. doi:10.1029/wr021i004p00563Hu, L. Y. (2000). Mathematical Geology, 32(1), 87-108. doi:10.1023/a:1007506918588Hu, L. Y., & Chugunova, T. (2008). Multiple-point geostatistics for modeling subsurface heterogeneity: A comprehensive review. Water Resources Research, 44(11). doi:10.1029/2008wr006993Huysmans, M., & Dassargues, A. (2009). Application of multiple-point geostatistics on modelling groundwater flow and transport in a cross-bedded aquifer (Belgium). Hydrogeology Journal, 17(8), 1901-1911. doi:10.1007/s10040-009-0495-2Jafarpour, B., & Khodabakhshi, M. (2011). A Probability Conditioning Method (PCM) for Nonlinear Flow Data Integration into Multipoint Statistical Facies Simulation. Mathematical Geosciences, 43(2), 133-164. doi:10.1007/s11004-011-9316-yJournel, A., & Zhang, T. (2006). The Necessity of a Multiple-Point Prior Model. Mathematical Geology, 38(5), 591-610. doi:10.1007/s11004-006-9031-2Kerrou, J., Renard, P., Hendricks Franssen, H.-J., & Lunati, I. (2008). Issues in characterizing heterogeneity and connectivity in non-multiGaussian media. Advances in Water Resources, 31(1), 147-159. doi:10.1016/j.advwatres.2007.07.002Kwa, Franz, M. O., & Scholkopf, B. (2005). Iterative kernel principal component analysis for image modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(9), 1351-1366. doi:10.1109/tpami.2005.181Kitanidis, P. K. (2007). On stochastic inverse modeling. Geophysical Monograph Series, 19-30. doi:10.1029/171gm04Kitanidis, P. K., & Vomvoris, E. G. (1983). A geostatistical approach to the inverse problem in groundwater modeling (steady state) and one-dimensional simulations. Water Resources Research, 19(3), 677-690. doi:10.1029/wr019i003p00677Li, L., Zhou, H., Hendricks Franssen, H. J., & Gómez-Hernández, J. J. (2011). Groundwater flow inverse modeling in non-MultiGaussian media: performance assessment of the normal-score Ensemble Kalman Filter. Hydrology and Earth System Sciences Discussions, 8(4), 6749-6788. doi:10.5194/hessd-8-6749-2011Li, L., Zhou, H., & Gómez-Hernández, J. J. (2011). Transport upscaling using multi-rate mass transfer in three-dimensional highly heterogeneous porous media. Advances in Water Resources, 34(4), 478-489. doi:10.1016/j.advwatres.2011.01.001Li, L., Zhou, H., & Gómez-Hernández, J. J. (2011). A comparative study of three-dimensional hydraulic conductivity upscaling at the macro-dispersion experiment (MADE) site, Columbus Air Force Base, Mississippi (USA). Journal of Hydrology, 404(3-4), 278-293. doi:10.1016/j.jhydrol.2011.05.001Mariethoz, G., Renard, P., & Straubhaar, J. (2010). The Direct Sampling method to perform multiple-point geostatistical simulations. Water Resources Research, 46(11). doi:10.1029/2008wr007621Mariethoz, G., Renard, P., & Caers, J. (2010). Bayesian inverse problem and optimization with iterative spatial resampling. Water Resources Research, 46(11). doi:10.1029/2010wr009274Neuman, S. P. (1973). Calibration of distributed parameter groundwater flow models viewed as a multiple-objective decision process under uncertainty. Water Resources Research, 9(4), 1006-1021. doi:10.1029/wr009i004p01006Oliver, D. S., Cunha, L. B., & Reynolds, A. C. (1997). Markov chain Monte Carlo methods for conditioning a permeability field to pressure data. Mathematical Geology, 29(1), 61-91. doi:10.1007/bf02769620RamaRao, B. S., LaVenue, A. M., De Marsily, G., & Marietta, M. G. (1995). Pilot Point Methodology for Automated Calibration of an Ensemble of conditionally Simulated Transmissivity Fields: 1. Theory and Computational Experiments. Water Resources Research, 31(3), 475-493. doi:10.1029/94wr02258Rubin, Y., Chen, X., Murakami, H., & Hahn, M. (2010). A Bayesian approach for inverse modeling, data assimilation, and conditional simulation of spatial random fields. Water Resources Research, 46(10). doi:10.1029/2009wr008799Strebelle, S. (2002). Mathematical Geology, 34(1), 1-21. doi:10.1023/a:1014009426274Suzuki, S., & Caers, J. (2008). A Distance-based Prior Model Parameterization for Constraining Solutions of Spatial Inverse Problems. Mathematical Geosciences, 40(4), 445-469. doi:10.1007/s11004-008-9154-8Wen, X. H., Capilla, J. E., Deutsch, C. V., Gómez-Hernández, J. J., & Cullick, A. S. (1999). A program to create permeability fields that honor single-phase flow rate and pressure data. Computers & Geosciences, 25(3), 217-230. doi:10.1016/s0098-3004(98)00126-5Zhang, T., Switzer, P., & Journel, A. (2006). Filter-Based Classification of Training Image Patterns for Spatial Simulation. Mathematical Geology, 38(1), 63-80. doi:10.1007/s11004-005-9004-xZhou, H., Gómez-Hernández, J. J., Hendricks Franssen, H.-J., & Li, L. (2011). An approach to handling non-Gaussianity of parameters and state variables in ensemble Kalman filtering. Advances in Water Resources, 34(7), 844-864. doi:10.1016/j.advwatres.2011.04.014Zhou, H., Li, L., Hendricks Franssen, H.-J., & Gómez-Hernández, J. J. (2011). Pattern Recognition in a Bimodal Aquifer Using the Normal-Score Ensemble Kalman Filter. Mathematical Geosciences, 44(2), 169-185. doi:10.1007/s11004-011-9372-

Cardiovascular adverse events in modern myeloma therapy - incidence and risks. A review from European Myeloma Network (EMN) and Italian Society of Arterial Hypertension (SIIA)

Cardiovascular disease in myeloma patients may derive from factors unrelated to the disease (age, diabetes, dyslipidemia, obesity, prior cardiovascular diseases), related to the disease (cardiac AL-amyloidosis, hyperviscosity, high-output failure, arteriovenous shunting, anemia, renal dysfunction) and linked to antimyeloma treatment (anthracyclines, corticosteroids, alkylating agents, immunomodulatory drugs, proteasome inhibitors). An accurate knowledge of cardiovascular events, effective dose reductions, prevention and management of early and late cardiovascular side effects of chemotherapeutic agents are essential in current clinical practice. Myeloma experts are obliged to carefully balance drugs' efficacy and toxicity for each individual patient. This review summarizes current data and novel insights on cardiovascular adverse events of today's antimyeloma treatment, focusing on carfilzomib, which is the starting point to develop consensus recommendations on preventing and managing cardiovascular side effects in myeloma patients